I've been doing a test on my local set up fetching tag values for load_generator.seq_num, this is a unique ID added to every trace by the synthetic-load-generator. Before these changes the amount of values always got truncated at about ~200 values. With this PR amount of values returned increases up to 80k and drops back to 20k when a block was flushed.

Avg latency sits around 200-400 ms.

TODO

• Check if the context has been cancelled
  
• Experiment with filtering tag values on service.name Next PR
• Most time is spent searching live traces, can we speed this up?

Still to address: when fetching tag values with very high cardinality (for instance http.url) requests fail/time out when being sent between components. I.e. the ingester is fine digging up all the values, but the GRPC connection between the ingester, querier and query-frontend was hitting the max bytes size...
We should add some limits (optionally configurable) to protect against this happening. Ideally, we return early when we have 'a lot of' values.

Future improvements (not for this PR): we can add filter capabilities to the tag values endpoint, so if someone starts typing http.url=/myService/ we only return values containing /myService/. This would make high cardinality tags more useable.

Remove the TagCache struct?

Yep 👍, I've removed tempodb/search/tag_cache.go, it's all the way at the bottom of the diff
https://github.com/grafana/tempo/pull/1068/files#diff-b29b479cf004a8bf08fb55114702b94f6d818911fca0c83c985e771db2fbe4bb